Enhancing signature word detection in voice assistants

ABSTRACT

Systems and methods detecting a spoken sentence in a speech recognition system are disclosed herein. Speech data is buffered based on an audio signal captured at a computing device operating in an active mode. The speech data is buffered irrespective of whether the speech data comprises a signature word. The buffered speech data is processed to detect a presence of the sentence comprising at least one command and a query for the computing device. Processing the buffered speech data includes detecting the signature word in the buffered speech data, and in response to detecting the signature word in the speech data, initiating detection of the sentence in the buffered speech data.

BACKGROUND

The present disclosure relates to speech recognition systems and, moreparticularly, to systems and methods related to speech-assisted deviceswith signature word recognition.

SUMMARY

Smart voice-assisted devices, smart devices commanded to perform certaintasks, are now ubiquitous to modern households and the commercialsector. The utterance of a signature word or phrase signals the deviceof a command or a query intended for the device to perform. The phrase“Ok, Google, play Game of Thrones!”, when spoken clearly into aGoogle-manufactured voice-assisted system, is commonly known to causethe device to carry out the user command to play the television series“Game of Thrones” on a media player, for example. Similarly, uttering“Alexa, please tell me the time!” causes a properly configuredspeech-recognition device, such as the Amazon Echo, to announce thecurrent time. Both “Ok, Google” and “Alexa”, spoken within an acceptablerange of a corresponding properly configured device, trigger a devicereaction. But in the absence of a signature word, and particularly asignature word that precedes each user command or query, the devicefails to take the commanded action and instead performs no response. Thevoice-assisted device is effectively deaf to a user command without apreceding signature word. The signature word is therefore key to theoperation of voice-assisted devices. What is perhaps even more key tothe proper operation of such devices is the order in which the signatureword appears in the spoken command or query. That is, what grabs theattention of a smart voice-assisted device to carry out a user-voicedcommand, e.g., “Play Game of Thrones” or “Please tell me the time,” isnot only a signature word but also the utterance of the signature wordin a predefined order, immediately before the spoken command, astructured and rather rigid approach to proper processing of a usercommand.

Repeating a signature word before uttering a command or query may seemsomewhat burdensome or unnatural for some users. It is rather atypical,for instance, for a friend to call a person by their name each timebefore uttering a sentence directed to the friend. “Jack, please stopwatching tv,” followed by “Jack, please get my bag from the table,”followed by “Jack, let's go” sounds awkward and unusual. Speaking asignature word in the beginning, middle or the end of a query or commandshould serve no consequence, yet, in today's devices, it does.

It is no secret that voice-assisted devices raise privacy concerns bycapturing vast amounts of recognizable and private communication spokenwithin a speaking range of the device. Long before a signature word,such as “Ok, Google,” “Alexa” or “TIVO,” is detected, all surroundingconversations are locally or remotely recorded. Moreover, certainprivacy regulations remain unaddressed. Absent proper user consent, anentire household of speech and conversation, over a span of numerousdays, weeks, months, and in many cases years, are unnecessarily andintrusively recorded and made available to a remotely located devicemanufacturer, completely removed from user control. Worse yet, manyusers remain ignorant of voice-assisted data collection privacyviolations. Recent privacy law enactments, in Europe, California, andBrazil, for example, demand manufacturers to place privacy rights oftheir users front and center by requiring express user consent beforeuser data collection, a condition not readily met by current-day smartdevices.

Accordingly, a less stringent and less intrusive electronic voiceassistant device, one without a strict pre-command signature wordrequirement and with a more natural user communication protocol, wouldbetter serve a voice-assistant user. In accordance with various speechrecognition embodiments and methods disclosed herein, a user eventindicative of a user intention to interact with a speech recognitiondevice is detected. In response to detecting the user event, an activemode of the speech recognition device is enabled to record speech databased on an audio signal captured at the speech recognition deviceirrespective of whether the speech data comprises a signature word.While the active mode is enabled, a recording of the speech data isgenerated, and the signature word is detected in a portion of the speechdata other than a beginning portion of the speech data. In response todetecting the signature word, the recording of the speech data isprocessed to recognize a user-uttered phrase.

In some embodiments, a method of detecting a sentence includes at leastone of a command and a query in a speech recognition system. Speech datais buffered based on an audio signal captured at a computing deviceoperating in an active mode. The speech data is buffered irrespective ofwhether the speech data comprises a signature word. The buffered speechdata is processed to detect the presence of a sentence comprising atleast one command and the query for the computing device. Processing thebuffered speech data includes detecting the signature word in thebuffered speech data, and, in response to detecting the signature wordin the speech data, initiating detection of the sentence in the bufferedspeech data.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the disclosure will beapparent upon consideration of the following detailed description, takenin conjunction with the accompanying drawings, in which:

FIGS. 1-2 each show an illustrative block diagram of a distinct speechrecognition system, in accordance with some embodiments of thedisclosure;

FIG. 3 depicts an illustrative flowchart of a speech recognitionprocess, in accordance with some embodiments of the disclosure;

FIG. 4 depicts an example speech detection technique, in accordance withsome embodiments of the disclosure;

FIG. 5 depicts an illustrative flowchart of a speech recognitionprocess, in accordance with some embodiments of the disclosure;

FIG. 6 depicts an illustrative flowchart of a speech recognitionprocess, in accordance with some embodiments of the disclosure;

FIG. 7 is a block diagram of an illustrative user device, in accordancewith some embodiments of the present disclosure; and

FIG. 8 is a block diagram of an illustrative system for transmittinginformation, in accordance with some embodiments of the presentdisclosure.

DETAILED DESCRIPTION

FIG. 1 shows an illustrative block diagram of speech recognition system100, in accordance with some embodiments of the present disclosure.System 100 is shown to include a speech recognition device 102communicatively coupled to a communication network 104, in accordancewith various disclosed embodiments. Speech recognition device 102 isshown to include an active mode buffer 116, a user activity detector 118and an audio signal receiver 120. Communication network 104 is shown toinclude a speech recognition processor 124. In some embodiments, speechrecognition device 102 may be implemented, in part or in whole, inhardware, software, or a combination of hardware and software. Forexample, a processor (e.g., control circuitry 704 of FIG. 7 ) executingprogram code stored in a storage location, such as storage 708 of FIG. 7, may perform, in part or in whole, some of the speech recognitionfunctions of device 102 disclosed herein. Similarly, speech recognitionprocessor 124 may be implemented, in part or in whole, in hardware,software, or a combination of hardware and software. For example, aprocessor (e.g., control circuitry 704 of FIG. 7 ) executing programcode stored in a storage location, such as storage 708 of FIG. 7 , mayperform, in part or in whole, some of the speech recognition functionsof processor 124 disclosed herein.

Communication network 104 may be a wide area network (WAN), a local areanetwork (LAN), or any other suitable network system. Communicationnetwork 104 may be made of one or multiple network systems. In someembodiments, communication network 104 and device 102 arecommunicatively coupled by one or more network communication interfaces.In some example systems, communication network 104 and device 102 arecommunicatively coupled by the interfaces shown and discussed relativeto FIG. 7 . Communication network 104 and device 102 may becommunicatively coupled in accordance with one or more suitable networkcommunication interfaces.

In accordance with an embodiment, speech recognition device 102 receivesaudio signals at audio signal receiver 120, processes the received audiosignals locally for speech recognition, and transmits the processedaudio signals to communication network 104 for further speechrecognition processing. For example, speech recognition device 102 mayreceive audio signals 110 and 114 from each of users 106 and 112,respectively, process the received signals 110 and 114 for speechprocessing with user activity detector 118 and active mode buffer 116and transmit the processed audio signals to speech recognition processor124 of communication network 104 for further voice recognitionprocessing. In some embodiments, processor 124 transmits the processedspeech file to a third-party transcription service for automated speechrecognition to translate voice into text and receive a text filecorresponding to the transmitted processed speech file. For example,processor 124 may send the processed speech file to Amazon Transcribeand Google Speech-to-Text.

In some embodiments, user activity detector 118 includes detecting andsensing components sensitive to recognizing a physical change related tothe user, such as, but without limitation, a physical user movementcloser in proximity to speech recognition device 102. For example, user106 may make a sudden physical head turn from a starting position 106 anot directly facing the audio signal receiver 120 of device 102, to aturned position 106 b, directly facing the audio signal receiver 120 ofdevice 102. To user activity detector 118, the detected user 106 turnaction signals a soon-to-follow audio signal 110 with a command or anassertion speech originating from user 106 or from the direction of user106. In contrast, in the absence of a physical change in user 112,activity detector 118 detects no user activity, user movement or audiostrength change, from user 112 or from the direction of user 112 thatmay suggest user 112 is possibly interested in interacting with device102.

User activity detector 118 may detect a user event in a variety of ways.For example, user activity detector 118 may implement a motion detectionfunction, using a motion detector device, to sense user 106 turn motionfrom position 106 a to position 106 b. Activity detector 118 mayalternatively or in combination implement a spectral analysis technique,using a spectral analyzer device, to detect an increased audio signalamplitude when receiving audio signal 110, corresponding to user 106, asuser 106 turns from position 106 a to position 106 b, directly facingaudio signal receiver 120 of device 102. Still alternatively or incombination, activity detector 118 may implement an image capturingfunction, using an image capturing device such as, without limitation, adigital camera, that captures images showing the user 106 turn movementfrom position 106 a to position 106 b. Device 102 may employ anysuitable technique using a corresponding suitable component that helpsdetect a closer proximity of user 106 to device 102. In the non-activemode where device 102 is waiting to detect a user movement, such asdiscussed above, device 102 remains in a continuous intimation detectionmode with functionality limited, in large part, to the detection with areduced power consumption requirement. In response to a detected useractivity, device 102 enables an active mode.

In the active mode, device 102 may start to record incoming audiosignals, such as signal 110, in a storage location, such as storage 708(FIG. 7 ). Audio signal 110 is made of audio/speech chunks, packets ofspeech data. In some embodiments, device 102 saves the speech datapackets in the active mode, in active mode buffer 116. Buffer 116 may bea part of or incorporated in storage 708 (FIG. 7 ). Audio signalreceiver 120 may be a microphone internally or externally locatedrelative to device 102.

In accordance with an example operational application, device 102 is aTIVO voice-enabled product. As depicted in FIG. 1 , at 1), user activitydetector 118 senses the user 106 turn movement from position 106 a toposition 106 b and, in response to detecting the user turn, device 102enables its active mode. While in the active mode, device 102 starts torecord incoming user utterances in the form of packets of speech dataand, at 2), looks for a signature word in the incoming speech datapackets. Device 102 stores the incoming speech data packets in activemode buffer 116, a local storage location. At 3), in response todetecting the signature word, for example, signature word “TIVO,” in auser 106 utterance, i.e., “Please tell me the time, TIVO!”, device 102begins a processing phase by transmitting the recorded speech datapackets, in the form of an audio file, from buffer 116 to communicationnetwork 104. Detection of the signature word, “TIVO,” at 3) in FIG. 1 ,effectively starts the processing of the received speech data packets.At communication network 104, the transmitted packets are processed torecognize the user utterance “Please tell me the time, TIVO!”, as shownat 4) in FIG. 1 . As used herein, the term “signature word” refers to aword, phrase, sentence, or any other form of utterance that addresses asmart assistance device.

In some embodiments, recording, prompted by a user activity as discussedabove, continues even after transmission and processing of the packetsbegins at communication network 104. In some embodiments, recordingstops in response to packet transmission to and processing bycommunication network 104.

As earlier noted, device 102 records user 106 utterances locally withoutsharing the recorded information with communication network 104 forprivacy reasons. User speech is therefore maintained confidentiallyuntil a signature word detection. In the case where no signature word isdetected, no recording of user utterances is generated. In someembodiments, in furtherance of user privacy protection, prior tostarting to generate a recording, device 102 may request a privacyconsent (e.g., consent to the collection of user speech) confirmationfrom user 106 and may further condition the recording on receiving theconsent. That is, device 102 simply does not record user utterances evenin the presence of a signature word detection unless a user consentacknowledgement is received. For example, device 102 may generate adisplay on a user device, such as a user smartphone or a user tablet,with privacy terms to be agreed to by the user. Device 102 may wait toreceive a response from the user acknowledging consent to the terms by,for example, clicking a corresponding box shown on the user devicedisplay.

In some embodiments, device 102 encrypts speech data packetscorresponding to user 106 utterances, for example, utterance “Pleasetell me the time, TIVO!”, before storing or recording the packets inbuffer 116, as yet another added security measure to ensure meetingstringent legal privacy requirements.

In accordance with some embodiments, the signature word, “TIVO,” isdetected despite its location in the user-uttered phrase. “TIVO” mayappear in the beginning, middle, end, or anywhere in between, in thephrase “Please tell me the time” yet be recognized in accordance withsome disclosed embodiments and methods. For example, the user 106 turn(from 106 a to 106 b) sets off a recording session guaranteeingpreservation of the signature word despite the signature word locationin the phrase.

As previously indicated, the speech data packets may be saved in asingle and local physical buffer with no other storage locationnecessitated, in part, because pre-active mode recording is unnecessary.This single buffer approach is yet another effective device 102energy-conservation measure.

FIG. 2 shows an illustrative block diagram of speech recognition system200, in accordance with some embodiments of the present disclosure. Inan example embodiment, as discussed below, system 200 is configured assystem 100 of FIG. 1 with further processing features shown anddiscussed relative to FIG. 2 .

System 200 is shown to include a speech recognition device 202communicatively coupled with a communication network 204. With continuedreference to the operational example of FIG. 1 , in FIG. 2 , an activitydetector 218 of device 202 detects a turn motion from position 206 a toposition 206 b by user 206 and, in response to the detection, device 202enables the active mode. In the active mode, device 202 records incomingspeech data packets corresponding to the user utterance “Please tell methe time, TIVO!”, in active mode buffer 216. Analogous to the example ofFIG. 1 , in FIG. 2 , device 102 stores at least speech data packetscorresponding to three phrases 234, namely phrases 1, 2, and 3 (234 a,234 b, and 234 c), originating from user 206, in buffer 216. The phrasesare stored in an audio file 230 in buffer 216. Audio buffer 230 may havea different number of phrases than that shown and discussed herein.

Audio file 230 further includes silent durations 232, each of which(silent duration 232 a, silent duration 232 b, and silent duration 232c) is located between two adjacent phrases in audio file 230. In someembodiments, device 102 performs some or all audio file processinglocally. For example, device 102 may perform detection and recognitionof a sentence, as disclosed herein, locally. In some embodiments, device102 and a speech recognition processor 224 of communication network 204share the tasks. In yet another embodiment, device 202 transmits audiofile 230 to communication network 204 for processing by processor 224,as discussed in large part relative to FIG. 1 . The discussion of FIG. 2to follow presumes the last scenario with device 202 transmitting audiofile 230 for processing by communication network 204.

In some embodiments, device 202 transmits audio file 230 tocommunication network 204 as buffer 216 becomes full, on a rollingbasis. In this connection, in accordance with some embodiments, buffer216 is presumed adequately large to accommodate at least a phrase worthof speech data packets. In some embodiments, device 202 transmits lessthan a buffer full of phrases to communication network 204. Forinstance, device 202 may transmit one, two, or three phrases as theybecome available in buffer 216 to communication network 204. In thisscenario, device 202 is equipped with the capability to detect thebeginning and ending of a phrase. In some embodiments, device 202 maydetect silent durations 232 to attempt to distinguish or parse asentence.

In some embodiments, as speech data packets are received at an audiosignal receiver 220 of device 202, device 202 may implement or solicit aspeech detection algorithm to determine the start and end of a phrasebased on a sequence validating technique. For example, device 202 mayimplement a segmental conditional random field (CRF) algorithm or use ahidden Markov model (HMM) or a long short-term memory (LSTM) model topredict the end of the audio signal corresponding to a phrase orsentence (or the beginning of a silent duration 232 in FIG. 2 ). Inimplementations using model-based prediction, such as with the use ofHMM or LSTM models, the model is trained to predict whether the utteredword is a start of the sentence, an intermediate word or the last wordof the sentence. As further described relative to FIG. 4 , a model istrained with and can therefore predict features such as, withoutlimitation, question tags, WH (“what”) words, articles, part-of-speechtags, intonations, syllables, or any other suitable language attributes.The term “tag,” as used herein, refers to a label that is attached to,stored with, or otherwise associated with a word or a phrase. Forinstance, “verb” is an example of a part-of-speech tag that may beassociated with the word “running.” As used herein, the term “feature”refers to a collection of different types of tag values. Part-of-speechis one example of a feature or a type of tag value. An influential wordis another example of a feature or a type of tag value. During thetraining of the model, a collection of word-to-tag mappings is fed tothe model along with an input sentence. As used herein, the term “label”refers to a value or outcome that corresponds to a sample input (e.g., aquery, features, or the like) and that may be employed during trainingof the model. In some examples, the model is trained by way ofsupervised learning based on labeled data, such as sample inputs andcorresponding labels. In some examples, features may be referred to asdependent variables, and labels may be referred to as independentvariables.

A sequence validation technique may be executed on a sentence or phrasein a forward and a backward direction for improved predictionreliability but at the expense of requiring a separate model and modeltraining for each direction, a rather costly approach. A sequencestructure validation may be employed using conditional probability atits base, for example, the Bayes theorem, to store states at differentpoints in time of a sentence. In some embodiments, an extension to thebasic sequence structure validation algorithm may be implemented withMarkov chains. Markov chains introduce hidden states at every statetransition, for example, between the words of a phrase or sentence, orbetween syllables of words of a phrase or sentence. The labels used foreach such training example are the points in time at which the phrase(spoken utterance) may start and end.

In some embodiments, the start of a phrase is typically driven bydecisions taken during the handling of the last packet of a phrase, anda list of contextual information is passed to the next audio chunk (orpacket). In some cases, a silent duration of a predefined duration maybe detected in real time to help shift to a new context. In someembodiments, silent duration detection may be implemented based onheuristics. For example, heuristics of reconfigurable manufacturingsystems (RMS) values representing speech data amplitude may be processedto detect silent durations in an audio file, such as the audio file 230of FIG. 2 .

In implementations with communication network 204 facilitating packetprocessing, processor 224 may achieve phrase detection by implementingthe foregoing speech detection algorithms described with reference todevice 202. For example, in an instance of audio file 230, audio file230′, shown at processor 224 of communication network 204 in FIG. 2 ,silent duration 232′ (232 a′, 232 b′, and 232 c′) may be detected toisolate or distinguish each of the phrases 234′ (234 a′, 234 b′, and 234c′). In the example of FIG. 2 , phrase 2, 234 b′ is shown detected atprocessor 224.

FIG. 3 shows an illustrative flowchart of a speech recognition process300, in accordance with some embodiments of the disclosure. Process 300may be performed, partially or in its entirety, by a voice-assisteddevice, such as devices 102 and 202 of FIGS. 1 and 2 , respectively. Insome embodiments, process 300 may be performed by control circuitry 704(FIG. 7 ). In some embodiments, process 300 may be performed locally orremotely or a combination thereof. For example, process 300 may beperformed, partially or in its entirety, by processor 124 or processor224 of FIGS. 1 and 2 , respectively. Process 300 may be performed by acombination of a voice-assisted device and a remote process, forexample, device 102 and processor 124 or device 202 and processor 224.

At 302, process 300 begins, and at step 304, a device implementingprocess 300 waits for the detection of a user event, such as a usermovement, as previously discussed. In response to the detection of auser event at step 304, process 300 proceeds to step 306, and an activemode of the device is enabled to start generating a recording of theincoming speech data packets. Next, at step 308, the speech data isrecorded and process 300 proceeds to step 310. At step 310, the deviceimplementing process 300 looks for a signature word in the recordedspeech data. In response to the detection of a signature word at step310, process 300 proceeds to step 312, and at step 312, the recordedspeech data is processed as described in accordance with variousdisclosed methods. For example, the recorded speech data may betransmitted to a network cloud device for processing. After step 312,process 300 resumes starting at step 304 to look for the next userevent. At step 304, a device implementing process 300 waits to detect auser event before proceeding to step 306, and in some embodiments, thedevice may abandon waiting for detection in response to a time outperiod or in response to a manual intervention, for example, by a userdevice.

As earlier noted, in some embodiments, at a communication network or avoice-enabled device, such as, without limitation, communicationnetworks 104, 204 and devices 102, 202, respectively, a model may betrained with various sentence features. For example, the model may betrained with the earlier-enumerated language attributes. Once the modelhas been trained, devices 102, 202 may utilize the model to generatelanguage attributes for a given sequence of inputted utterances. FIG. 4shows an example table 400 of an output that devices 102, 202 maygenerate by employing one or more speech detection techniques oralgorithms upon a sequence of utterances, in accordance with somedisclosed embodiments. In some aspects, the utterance (or sentence)structure features shown in FIG. 4 may be used to train a model ofvarious disclosed embodiments and methods.

Example types of algorithms that devices 102, 202 may employ include,without limitation, algorithms that determine whether each term in aquery is a “WH” term (e.g., based on text generated from theutterances), determine whether each term in the query is an article(e.g., “a” or “the”), determine a part-of-speech for each term of thequery, and determine the syllables of each term in the query. In someexamples, the “WH” terms and article detection may be performed byprocessing text strings that are generated from the utterances. Exampleparts of speech algorithms that devices 102, 202 may employ, forinstance, include those that are provided by the Natural LanguageToolkit (NLTK), spaCy, and/or other natural language processingproviders. Some of such algorithms train parts of speech models usingclassifiers such as DecisionTree, vectorizers, and/or the like. In oneexample, syllables are extracted from utterances by using a raw audiosignal to detect multiple audio features and voice activity.Praat/Praat-Parselmouth is one example of an open source tool kit thatmay be employed for such syllable extraction. In another example, anAncient Soundex algorithm can extract syllables from utterances by usingtext generated based on the utterances. Metaphone, Double metaphone, andMetaphone-3 are example algorithms that may perform text-based syllableextraction.

Table 400 includes columns 404 with each column including a word of thephrase “What is the time, TWO?”, for example, uttered by user 106 oruser 206 of FIGS. 1 and 2 , respectively. Table 400 further includesrows 402, with each row representing a tag or a training feature. Forexample, the first row is for the feature “WH,” the second row is forthe feature “articles,” the third row is for the feature “POS” and thefourth row is for the feature “syllables.” An acoustic model may betrained with a set of features that are in part or in whole differentthan the feature set of FIG. 4 , or the model may be trained with afeature set that includes less than four or more than four features. Ingeneral, the greater the number of sentence features the model trainswith, the greater the accuracy of sentence prediction.

Table 400 entries are marked based on the feature corresponding to eachword of the sentence “What is the time, TIVO?”. For example, “What”corresponds to the feature “WH” but the word “is” or the word “the” or“time” do not. Accordingly, a checkmark is placed in the entry of table400 at the first row and first column. Similarly, the word “the” is anarticle and marked accordingly in the second row, third column of Table400 and so on. In this respect, an acoustic model is trained to predictthe words of a sentence and therefore the entire sentence. In apractical example, the model may be used to predict the words of asentence at step 312 of process 300 (FIG. 3 ) and step 510 of FIG. 5 .

FIG. 5 shows an illustrative flowchart of a speech recognition process,in accordance with some embodiments of the disclosure. In FIG. 5 , aprocess 500 may be performed by a voice-assisted device, such as devices102 and 202 of FIGS. 1 and 2 , respectively, to process incoming speechdata packets. In some embodiments, the steps of process 500 may beperformed by control circuitry 704 of FIG. 7 . In summary, process 500presents an example of a method for detecting a spoken sentence in aspeech recognition system as disclosed herein. Speech data is bufferedbased on an audio signal captured at a control circuitry operating in anactive mode. The speech data is buffered irrespective of whether thespeech data comprises a signature word. The buffered speech data isprocessed to detect the presence of a sentence comprising at least onecommand and a query for the computing device. Processing the bufferedspeech data includes detecting the signature word in the buffered speechdata, and, in response to detecting the signature word in the speechdata, initiating detection of the sentence in the buffered speech data.

More specifically and with reference to FIG. 5 , at 502, process 500starts and continues to step 504 where packets of speech data,corresponding to a user-spoken sentence, are buffered based on an audiosignal captured in an active mode, as earlier described. The packets arepreviously received, for example, at audio signal receiver 120 orreceiver 220 of devices 102 and 202, respectively. While in active mode,the received data packets may be recorded in buffer 116 or buffer 216 ofdevices 101 and 102, respectively. Next, at step 506, the bufferedspeech data packets are processed. The voice-assisted device, such asmay be implemented by control circuitry 704 (FIG. 7 ), detects thesignature word at step 508, followed, at step 510, by initiatingdetection of the sentence in the buffered speech data, in response todetecting the signature word at step 508, in step 510. Steps 508 and 510are part of the processing that starts at step 508. Processing isperformed while the device remains in active mode. In some embodiments,the device leaves the active mode in response to a manual configuration,such as in response to receiving a corresponding user device signal. Insome embodiments, the device may leave an active mode if a signatureword is not found during a predefined time period at step 508. In someembodiments, the device leaves the active mode in response to receivingspeech data packets corresponding to an entire spoken sentence.

FIG. 6 shows an illustrative flowchart of a speech recognition process,in accordance with some embodiments of the disclosure. In FIG. 6 , aprocess 600 may be performed by a remotely located (relative to acommunicatively coupled voice-assisted device) processor, such asprocessor 124 of FIG. 1 or processor 224 of FIG. 2 . Process 600 beginsat 602 and continues to step 604 where an audio file with recordedpackets of speech data corresponding to at least one spoken sentence isreceived. In FIG. 6 , the audio file is presumed to include N number ofpackets, “N” representing an integer value. In some embodiments, theaudio file of step 604 may be received from device 102 or device 202.Next, at step 606, the beginning and ending of the sentence in the audiofile of step 604 are identified. If, at step 608, process 600 determinesthat all N sentences of the audio file have been processed, process 600continues to step 604 and starts to process the next audio file after itis received as previously described. If, at step 608, process 600determines not all sentences of the audio file have been processed,process 600 proceeds to step 610. At step 610, the current sentence, thesentence identified at step 606, is processed and next, at step 612, theprocessing of the next sentence of the audio file begins, and the“current” sentence of the following steps in process 600, i.e., steps604 through 610, is the next sequential sentence in the audio file. Insome embodiments, phrases of an audio file need not be sequentiallyprocessed. For example, phrase 3 may be processed before phrase 2 inFIG. 2 . But in certain implementations using context speech recognitiontechniques, the accuracy of sentence prediction may improve if thesentences are sequentially processed.

At step 610, the current sentence may be transmitted to a remoteautomated speech recognition (ASR) service for text transcription. Insome embodiments, ASR services may be performed on the audio file afterall sentences of the file have been processed. In process 600, ASRservices are presumed performed on a sentence basis rather than on anaudio file basis.

The order of steps of each of the processes 300, 500 and 600, as shownin the flowcharts of FIGS. 3, 5, and 6 , respectively, may be suitablychanged or exchanged. One or more steps, as may be suitable, can beadded or deleted to each of the processes 300, 500 and 600.

A user may access, process, transmit and receive content, in addition toother features, for example to carry out the functions andimplementations shown and described herein, with one or more userdevices (i.e., user equipment). FIG. 7 shows generalized embodiments ofan illustrative user device. In some embodiments, user device 700 may beconfigured, in whole or in part, as a computing device. Althoughillustrated as a mobile user device (e.g., a smartphone), user device700 may include any user electronic device that performs speechrecognition operations as disclosed herein. In some embodiments, userdevice 700 may incorporate, in part or in whole, or be communicativelycoupled to, each of devices 102 and 202 of FIGS. 1 and 2 . In someembodiments, user device 700 may include a desktop computer, a tablet, alaptop, a remote server, any other suitable device, or any combinationthereof, for speech detection and recognition processing, as describedabove, or accessing content, such as, without limitation, wearabledevices with projected image reflection capability, such as ahead-mounted display (HMD) (e.g., optical head-mounted display (OHMD)),electronic devices with computer vision features, such as augmentedreality (AR), virtual reality (VR), extended reality (XR), or mixedreality (MR), portable hub computing packs, a television, a Smart TV, aset-top box, an integrated receiver decoder (IRD) for handling satellitetelevision, a digital storage device, a digital media receiver (DMR), adigital media adapter (DMA), a streaming media device, a DVD player, aDVD recorder, a connected DVD, a local media server, a BLU-RAY player, aBLU-RAY recorder, a personal computer (PC), a laptop computer, a tabletcomputer, a WebTV box, a personal computer television (PC/TV), a PCmedia server, a PC media center, a handheld computer, a stationarytelephone, a personal digital assistant (PDA), a mobile telephone, aportable video player, a portable music player, a portable gamingmachine, a smartphone, or any other television equipment, computingequipment, or wireless device, and/or combination of the same. In someembodiments, the user device may have a front-facing screen and arear-facing screen, multiple front screens, or multiple angled screens.In some embodiments, the user device may have a front-facing cameraand/or a rear-facing camera. On these user devices, users may be able tonavigate among and locate the same content available through atelevision. Consequently, a user interface in accordance with thepresent disclosure may be available on these devices, as well. The userinterface may be for content available only through a television, forcontent available only through one or more of other types of userdevices, or for content available both through a television and one ormore of the other types of user devices. The user interfaces describedherein may be provided as online applications (i.e., provided on awebsite), or as stand-alone applications or clients on user equipmentdevices. Various devices and platforms that may implement the presentdisclosure are described in more detail below.

In some embodiments, display 712 may include a touchscreen, a televisiondisplay or a computer display. In a practical example, display 712 maydisplay detected phrases from user utterances, as processed by devices102 and 202 or at communication networks 104 and 204. Alternatively, oradditionally, display 712 may show a respective user the terms of a userprivacy agreement, as previously discussed relative to FIGS. 1 and 2 .Display 712 may optionally show text results received from an ASRservice. In some embodiments, the one or more circuit boards illustratedinclude processing circuitry, control circuitry, and storage (e.g., RAM,ROM, Hard Disk, Removable Disk, etc.). In some embodiments, theprocessing circuit, control circuitry, or a combination thereof, mayimplement one or more of the processes of FIGS. 3, 5, and 6 . In someembodiments, the processing circuitry, control circuitry, or acombination thereof, may implement one or more functions or componentsof the devices of FIGS. 1 and 2 , such as devices 102 and 202, and/orprocessors 124 and 224. For example, each or a combination of activitydetector 118 or 218 and processor 124 or 224 of FIGS. 1 and 2 may beimplemented by the processing circuitry, control circuitry or acombination of the processing circuitry and control circuitry.

In some embodiments, circuit boards include an input/output path. Userdevice 700 may receive content and data via input/output (hereinafter“I/O”) path 702. I/O path 702 may provide content and data to controlcircuitry 704, which includes processing circuitry 706 and storage 708.Control circuitry 704 may be used to send and receive commands,requests, and other suitable data using I/O path 702. I/O path 702 mayconnect control circuitry 704 (and specifically processing circuitry706) to one or more communications paths (described below). I/Ofunctions may be provided by one or more of these communications pathsbut are shown as a single path in FIG. 7 to avoid overcomplicating thedrawing.

Control circuitry 704 may be based on any suitable processing circuitrysuch as processing circuitry 706. As referred to herein, processingcircuitry should be understood to mean circuitry based on one or moremicroprocessors, microcontrollers, digital signal processors,programmable logic devices, field-programmable gate arrays (FPGAs),application-specific integrated circuits (ASICs), etc., and may includea multi-core processor (e.g., dual-core, quad-core, hexa-core, or anysuitable number of cores) or supercomputer. In some embodiments,processing circuitry is distributed across multiple separate processorsor processing units, for example, multiple of the same type ofprocessing units (e.g., two Intel Core i7 processors) or multipledifferent processors (e.g., an Intel Core i5 processor and an Intel Corei7 processor). In some embodiments, control circuitry 704 executesinstructions for an application stored in memory (e.g., storage 708).Specifically, control circuitry 704 may be instructed by the applicationto perform the functions discussed above and below. For example, theapplication may provide instructions to control circuitry 704 to performspeech detection and recognition processes as described herein. In someimplementations, any action performed by control circuitry 704 may bebased on instructions received from the application.

In some client/server-based embodiments, control circuitry 704 includescommunications circuitry suitable for communicating with an applicationserver or other networks or servers. The instructions for carrying outthe above-mentioned functionality may be stored on the applicationserver. Communications circuitry may include a wired or wireless modemor an ethernet card for communications with other equipment, or anyother suitable communications circuitry. Such communications may involvethe Internet or any other suitable communications networks or paths. Inaddition, communications circuitry may include circuitry that enablespeer-to-peer communication of user equipment devices, or communicationof user equipment devices in locations remote from each other (describedin more detail below).

Memory may be an electronic storage device provided as storage 708 thatis part of control circuitry 704. As referred to herein, the phrase“electronic storage device” or “storage device” or “memory” should beunderstood to mean any device for storing electronic data, computersoftware, or firmware, such as random-access memory, read-only memory,hard drives, optical drives, solid state devices, quantum storagedevices, gaming consoles, gaming media, or any other suitable fixed orremovable storage devices, and/or any combination of the same. Storage708 may be used to store various types of content described herein aswell as media guidance data described above. Nonvolatile memory may alsobe used (e.g., to launch a boot-up routine and other instructions).Cloud-based storage, for example, may be used to supplement storage 708or instead of storage 708. In some embodiments, storage 708 mayincorporate, in part or in whole, buffer 116 and buffer 216 of FIGS. 1and 2 , respectively.

In some embodiments, display 712 is caused by generation of a display bydevices 102 and 202 of FIGS. 1 and 2 , respectively, or user devicescoupled to devices 102 and 202. A user may send instructions to controlcircuitry 704 using user input interface 710. User input interface 710,display 712, or both may include a touchscreen configured to provide adisplay and receive haptic input. For example, the touchscreen may beconfigured to receive haptic input from a finger, a stylus, or both. Insome embodiments, equipment device 700 may include a front-facing screenand a rear-facing screen, multiple front screens, or multiple angledscreens. In some embodiments, user input interface 710 includes aremote-control device having one or more microphones, buttons, keypads,any other components configured to receive user input or combinationsthereof. For example, user input interface 710 may include a handheldremote-control device having an alphanumeric keypad and option buttons.

Audio equipment 714 may be provided as integrated with other elements ofuser device 700 or may be stand-alone units. The audio component ofvideos and other content displayed on display 712 may be played throughspeakers of audio equipment 714. In some embodiments, the audio may bedistributed to a receiver (not shown), which processes and outputs theaudio via speakers of audio equipment 714. In some embodiments, forexample, control circuitry 704 is configured to provide audio cues to auser, or other audio feedback to a user, using speakers of audioequipment 714. Audio equipment 714 may include a microphone configuredto receive audio input such as voice commands or speech. For example, auser may speak letters or words that are received by the microphone andconverted to text by control circuitry 704. In a further example, a usermay voice commands that are received by the microphone and recognized bycontrol circuitry 704.

An application may be implemented using any suitable architecture. Forexample, a stand-alone application may be wholly implemented on userdevice 700. In some such embodiments, instructions for the applicationare stored locally (e.g., in storage 708), and data for use by theapplication is downloaded on a periodic basis (e.g., from an out-of-bandfeed, from an Internet resource, or using another suitable approach).Control circuitry 704 may retrieve instructions of the application fromstorage 708 and process the instructions to generate any of the displaysdiscussed herein. Based on the processed instructions, control circuitry704 may determine what action to perform when input is received frominput interface 710. For example, movement of a cursor on a displayup/down may be indicated by the processed instructions when inputinterface 710 indicates that an up/down button was selected. Anapplication and/or any instructions for performing any of theembodiments discussed herein may be encoded on computer-readable media.Computer-readable media includes any media capable of storing data. Thecomputer-readable media may be transitory, including, but not limitedto, propagating electrical or electromagnetic signals, or it may benon-transitory including, but not limited to, volatile and non-volatilecomputer memory or storage devices such as a hard disk, floppy disk, USBdrive, DVD, CD, media cards, register memory, processor caches, RandomAccess Memory (RAM), etc.

In some embodiments, the application is a client/server-basedapplication. Data for use by a thick or thin client implemented on userdevice 700 is retrieved on demand by issuing requests to a server remotefrom user device 700. For example, the remote server may store theinstructions for the application in a storage device. The remote servermay process the stored instructions using circuitry (e.g., controlcircuitry 704) and generate the displays discussed above and below. Theclient device may receive the displays generated by the remote serverand may display the content of the displays locally on user device 700.This way, the processing of the instructions is performed remotely bythe server while the resulting displays (e.g., that may include text, akeyboard, or other visuals) are provided locally on user device 700.User device 700 may receive inputs from the user via input interface 610and transmit those inputs to the remote server for processing andgenerating the corresponding displays. For example, user device 700 maytransmit a communication to the remote server indicating that an up/downbutton was selected via input interface 710. The remote server mayprocess instructions in accordance with that input and generate adisplay of the application corresponding to the input (e.g., a displaythat moves a cursor up/down). The generated display is then transmittedto user device 700 for presentation to the user.

FIG. 8 is a block diagram of illustrative system 800 for transmittingmessages, in accordance with some embodiments of the present disclosure.In system 800, there may be more than one of each type of user device,but only one of each is shown in FIG. 8 to avoid overcomplicating thedrawing. In addition, each user may utilize more than one type of userdevice and more than one of each type of user device.

User device 820, illustrated as a wireless-enabled device, may becoupled to communication network 802 (e.g., the Internet). For example,user device 820 is coupled to communication network 802 viacommunications path 822 to access point 824 and wired connection 826.User device 820 may also include wired connections to a LAN, or anyother suitable communications link to network 802. Communication network802 may be one or more networks including the Internet, a mobile phonenetwork, mobile voice or data network (e.g., a WIFI, WiMAX, GSM, UTMS,CDMA, TDMA, 3G, 4G, 4G, 5G, Li-Fi, LTE network), cable network, publicswitched telephone network, or other types of communication network orcombinations of communication networks. Path 812 may include one or morecommunications paths, such as a satellite path, a fiber-optic path, acable path, a path that supports Internet communications, a free-spaceconnection (e.g., for broadcast or other wireless signals), or any othersuitable wired or wireless communications path or combination of suchpaths.

System 800 includes network entity 804 (e.g., a server or other suitablecomputing device) coupled to communication network 802 viacommunications path 812. Communications with network entity 804 may beexchanged over one or more communications paths but are shown as asingle path in FIG. 8 to avoid overcomplicating the drawing. Networkentity 804 is configured to access database 806 or applications 808(e.g., an applications database or host server) via communications links814 and 816, respectively. Although shown as a separate device, networkentity 804 may include database 806 and applications 808 (e.g., storedin memory). In addition, there may be more than one of each of database806 and applications 808, but only one of each is shown in FIG. 8 toavoid overcomplicating the drawing. In some embodiments, database 806and applications 808 may be integrated as one source device (e.g., thatmay be, but need not be, network entity 804).

Database 806 may include one or more types of stored information,including, for example, relationship information, a relationship entitydatabase, recipient information, historical communications records, userpreferences, user profile information, a template database, any othersuitable information, or any combination thereof. Applications 808 mayinclude an applications-hosting database or server, plug-ins, a softwaredevelopers kit (SDK), an applications programming interface (API), orother software tools configured to provide software (e.g., as downloadto a user device); run software remotely (e.g., hosting applicationsaccessed by user devices); or otherwise provide applications support toapplications of user device 820. In some embodiments, information fromnetwork entity 804, database 806, applications 808, or a combinationthereof may be provided to a user device using a client/server approach.For example, user device 820 may pull information from a server, or aserver may push information to user device 820. In some embodiments, anapplication client residing on user device 820 may initiate sessionswith database 806, applications 808, network entity 804, or acombination thereof to obtain information when needed (e.g., when datais out-of-date or when a user device receives a request from the user toreceive data). In some embodiments, information may include userinformation. For example, the user information may include currentand/or historical user activity information (e.g., what communicationsthe user engages in, what times of day the user sends/receives messages,whether the user interacts with a social network, at what times the userinteracts with a social network to post information, what types ofcontent the user typically inserts in messages, stored contacts of theuser, frequent contacts of the user, any other suitable information, orany combination thereof. In some embodiments, the user information mayidentify patterns of a given user for a period of more than one year.

In some embodiments, an application may include an application programprocessor implementing some of the processes and methods disclosedherein as a stand-alone application implemented on user device 820. Forexample, the application may be implemented as software or a set ofexecutable instructions, which may be stored in storage (e.g., storage708) of the user device (e.g., user device 700), and executed by controlcircuitry (e.g., control circuitry 704) of the user device (e.g., userdevice 700). In some embodiments, an application may include anautomatic program retrieval application that is implemented as aclient/server-based application where only a client application resideson the user device, and a server application resides on a remote server(e.g., network entity 804). For example, an automatic program retrievalapplication may be implemented partially as a client application on userdevice 820 (e.g., by control circuitry 704 of user equipment device 700)and partially on a remote server as a server application running oncontrol circuitry of the remote server (e.g., control circuitry ofnetwork entity 804). When executed by control circuitry of the remoteserver, the automatic program retrieval application may instruct thecontrol circuitry to generate the displays and transmit the generateddisplays to user device 820. The server application may instruct thecontrol circuitry of the remote device to transmit data for storage onuser device 820. The client application may instruct control circuitryof the receiving user device to generate the application displays.

In some embodiments, the arrangement of system 800 is a cloud-basedarrangement. The cloud provides access to services, such as informationstorage, messaging, or social networking services, among other examples,as well as access to any content described above, for user devices.Services can be provided in the cloud through cloud computing serviceproviders, or through other providers of online services. For example,the cloud-based services can include a storage service, a sharing site,a social networking site, or other services via which user-sourcedcontent is distributed for viewing by others on connected devices. Thesecloud-based services may allow a user device to store information to thecloud and to receive information from the cloud rather than storinginformation locally and accessing locally stored information. Cloudresources may be accessed by a user device using, for example, a webbrowser, a messaging application, a desktop application, a mobileapplication, and/or any combination of the same access applications. Theuser device may be a cloud client that relies on cloud computing forapplication delivery, or the user equipment device may have somefunctionality without access to cloud resources. For example, someapplications running on the user device may be cloud applications (e.g.,applications delivered as a service over the Internet), while otherapplications may be stored and run on the user device. In someembodiments, a user device may receive information from multiple cloudresources simultaneously.

The systems and processes discussed above are intended to beillustrative and not limiting. One skilled in the art would appreciatethat the actions of the processes discussed herein may be omitted,modified, combined, and/or rearranged, and any additional actions may beperformed without departing from the scope of the invention. Moregenerally, the above disclosure is meant to be exemplary and notlimiting. Only the claims that follow are meant to set bounds as to whatthe present disclosure includes. Furthermore, it should be noted thatthe features and limitations described in any one embodiment may beapplied to any other embodiment herein, and flowcharts or examplesrelating to one embodiment may be combined with any other embodiment ina suitable manner, done in different orders, or done in parallel. Inaddition, the systems and methods described herein may be performed inreal time. It should also be noted that the systems and/or methodsdescribed above may be applied to, or used in accordance with, othersystems and/or methods.

What is claimed is:
 1. A method for detecting a sentence including atleast one of a command and a query in a speech recognition system, themethod comprising: detecting, while a computing device is in anon-active mode, a user activity that suggests a user intention tointeract with the computing device, wherein speech data is not bufferedby the computing device in the non-active mode; in response to detectingthe user activity, enabling an active mode and buffering speech databased on an audio signal captured at the computing device operating inthe active mode, wherein the speech data is buffered in the active modeirrespective of whether the speech data comprises a signature word; andprocessing the buffered speech data to detect a presence of the sentencecomprising at least one of the command and the query for the computingdevice, wherein processing the buffered speech data comprises: detectingthe signature word in the buffered speech data; in response to detectingthe signature word in the speech data, initiating detection of thesentence in the buffered speech data; and detecting the sentence byidentifying a beginning portion of the sentence in the buffered speechdata and determining that the beginning portion of the sentence precedesa portion of the sentence corresponding to the signature word.
 2. Themethod of claim 1, further comprising detecting the sentence based on asequence validating technique or based on a model trained to distinguishbetween user commands and user assertions.
 3. The method of claim 1,further comprising detecting the sentence by detecting silent durationsoccurring before and after, respectively, the sentence in the speechdata, wherein detecting the silent durations is based on speechamplitude heuristics of the speech data.
 4. The method of claim 1,wherein detecting the signature word is performed at the computingdevice or at a server remote from the computing device.
 5. The method ofclaim 1, further comprising transmitting the speech data to a speechrecognition processor for performing automated speech recognition (ASR)on the speech data.
 6. The method of claim 1, further comprisingidentifying the beginning portion of the sentence and an end portion ofthe sentence based on a trained model selected from one of a hiddenMarkov model (HMM), a long short-term memory (LSTM) model, and abidirectional LSTM.
 7. The method of claim 1, wherein detecting thesignature word is based on heuristics of audio signatures of ademographic region.
 8. The method of claim 1, wherein the computingdevice operates in the active mode only in response to receiving a userconsent.
 9. The method of claim 1, wherein detecting the user activitycomprises determining that a user has turned to face the computingdevice.
 10. A system for detecting a sentence including at least one ofa command and a query in a speech recognition system, the systemcomprising: a memory; and control circuitry communicatively coupled tothe memory and configured to: detect, while a computing device is in anon-active mode, a user activity that suggests a user intention tointeract with the computing device, wherein speech data is not bufferedby the computing device in the non-active mode; in response to detectingthe user activity, enable an active mode and buffer in the memory speechdata based on an audio signal captured at the computing device operatingin the active mode, wherein the speech data is buffered in the activemode irrespective of whether the speech data comprises a signature word;and process the buffered speech data to detect a presence of thesentence comprising at least one of the command and the query for thecomputing device, wherein in processing the buffered speech data, thecontrol circuitry is configured to: detect the signature word in thebuffered speech data; in response to detecting the signature word in thespeech data, initiate detection of the sentence in the buffered speechdata; and detect the sentence by identifying a beginning portion of thesentence in the buffered speech data and determining that the beginningportion of the sentence precedes a portion of the sentence correspondingto the signature word.
 11. The system of claim 10, wherein the controlcircuitry is further configured to detect the sentence based on asequence validating technique or based on a model trained to distinguishbetween user commands and user assertions.
 12. The system of claim 10,wherein the control circuitry is further configured to: detect thesignature word by detecting silent durations occurring before and after,respectively, the sentence in the speech data; and the detect the silentdurations based on speech amplitude heuristics of the speech data. 13.The system of claim 10, wherein the memory is local to the computingdevice.
 14. The system of claim 10, wherein the control circuitry isfurther configured to detect the signature word at the computing device.15. The system of claim 10, wherein the control circuitry is configuredto transmit the buffered data packets to a remotely located server todetect the signature word.
 16. The system of claim 10, wherein thecontrol circuitry is further configured to transmit the speech data to aspeech recognition processor for performing automated speech recognition(ASR) on the speech data.
 17. The system of claim 10, wherein thecontrol circuitry is further configured to identify the beginningportion of the sentence and an end portion of the sentence based on atrained model selected from one of a hidden Markov model (HMM), a longshort-term memory (LSTM) model, and a bidirectional LSTM.
 18. The systemof claim 10, wherein detection of the signature word is based onheuristics of audio signatures of a demographic region.
 19. The systemof claim 10, wherein the control circuitry is configured to operate inthe active mode only in response to receiving a user consent.
 20. Thesystem of claim 10, wherein detecting the user activity comprisesdetermining that a user has turned to face the computing device.